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(54) Image processing apparatus and method, and storage medium storing image processing 
program 



(57) Disclosed are an image processing apparatus 
and a method therefor, which are characterized by the 
input of image data, the generation of security data to 
protect said image data, the control of an encoding 



mode in accordance with said security data, and the en- 
coding of said image data using said encoding mode 
that is controlled. Said image processing is performed 
to protect a copyright for an image. 



FIG. 3 



1000 



SECURITY 




ENCODING 




UNIT 








1001 








VIDEO 




ENCODING 




UNIT 







1002 



1003 



MULTIPLEXER - 



1004 



COMMUNICATION 
1/F 




1006 



COMMUNICATION 
I/F 



1007 



SEPARAT0R 



SECURITY 
DECODING 
UNIT 



1008 
-A 



1009 



VIDEO 

DECODING h*- 
UNIT 



1010 



CM 
< 

CM 

O 

Q. 
LU 

Printed by Jouve, 75001 PARIS (FR) 

BNSDOCID: <EP 1014724A2J_> 



1 



EP1 014 724 A2 



2 



Description 

BACKGROUND OF THE INVENTION 
Field of the Invention 

[0001] The present invention relates to image 
processing performed to protect a copyright for an im- 
age, and the like. 

Related Background Art 

[0002] Recently, the standardization of motion image 
encoding techniques has been advanced rapidly And 
encoding standards, such as MPEG-1, MPEG-2, H.261 
and H.263 (these encoding systems are specifically ac- 
cepted as international standards by ISO (International 
Organization for Standardization) and ITU (International 
Telecommunication Union)), have now been estab- 
lished. Further, work is progressing on MPEG-4 : a new 
international standard that is intended to be a general- 
purpose, next-generation multimedia encoding stand- 
ard that can be used in many fields, such as computers, 
broadcasting and communication. Image digitization 
has also been advanced by the encoding technique, and 
the storage and communication of motion image data 
have been markedly improved. 

[0003] For example, with MP EG -4, a motion image 
scene at a specific moment is regarded as an aggregate 
of objects, such as a "person", a "car", a "background", 
"music" and "video", which are the components of the 
scene, and an appropriate encoding process is per- 
formed for each object to efficiently encode all the mo- 
tion image data. 

[0004] According to a mu It i -object encoding tech- 
nique such as MPEG-4, a frame for a scene is ex- 
pressed by combining individual objects that have been 
encoded and compressed. Therefore, object data ob- 
tained from specific MPEG-4 data can be re-used with 
different MPEG-4 data, so that another frame can be 
easily expressed. 

[0005] As the above described digital encoding stand- 
ards have spread, the contents business has strongly 
expressed the problem they represent for the provision 
of copyright protection. That is, if the standards do not 
satisfactorily ensure adequate copyright protection, 
safe and superior contents can not be provided. 
[0006] For MPEG-4 this is a very serious problem. 
This is so because, as is described above, with MPEG- 
4 it is possible to encode and to operate/process data 
for individual objects. Therefore, since the use of the 
multi-object encoding system MPEG-4 permits the 
seamless reuse of individual objects, it is natural for re- 
quests to be made that protection be provided for the 
intellectual property represented by an object. And if the 
target is a person, it is natural that protection be sought 
for the right of portrait. 

[0007] To protect a copyright for a portion of a motion 



image, a method is available for temporarily halting the 
decoding of the motion image to prevent the reproduc- 
tion of a part of it. According to this method, while taking 
into account the right of portrait and the copyright for the 
s motbn image, the decoding of a portion related to them 
is halted until that portion has been passed, at which 
time the decoding is resumed. 

[0008] With this method, however, the following prob- 
lem is encountered. 

10 [0009] The encoding system for using interframe cor- 
relation is generally employed to encode motion imag- 
es. In the previously mentioned, well-known encoding 
systems H.261 and H.263, and MPEG-1 , 2 and 4, basi- 
cally, the preceding frame, or both the preceding and 

is the succeeding frames, are referred to along the time 
axis, and motion compensation is performed to encode 
motbn images. 

[0010] Data reproduction provided by H.261 or H.263 
is shown in Fig. 1 . U denotes a frame lor intraframe en- 

20 coding, and P* denotes a frame for interframe encoding. 
In Fig. 1 , Time is the direction in which time elapses, and 
in Security, a period during which the decoding process 
is halted is represented as a black block. Code repre- 
sents the arrangement of screens in the order in which 

25 encoded, and Display represents the arrangement of 
screens in the order in which displayed. 
[0011] Suppose that the decoding of the encoded da- 
ta for frames P 4 to P 9 is halted to apply the protection 
(Security) provided for an intellectual property (e.g., a 

30 copyright). The decoding of motion images is halted at 
data P 3 , and images are not again displayed until the 
decoding is resumed. Since the writing of encoded data 
to a buffer is also halted at the same time, the encoded 
data from P 4 to P 9 are abandoned. Therefore, when the 

35 image protection time has ended and the decoding is 
resumed, data P 9 , which originally should have been re- 
ferred to by data P 10 , have already been abandoned, so 
that data P 10 and the following data can not be normally 
decoded. As a result, until frame \ A , for which intraframe 

40 encoding is performed, is decoded, in an interval ex- 
tending from P 10 to P 13 , the images in the frames are 
deteriorated, or the decoding process is halted. 
[0012] MPEG-1, 2, and 4 also employ bidirectional 
motion compensation, whereby motion along the time 

45 axis is predicted from preceding and succeeding 
frames. 

[0013] The data reproduction provided by MPEG-1, 
2, and 4 is shown in Fig. 2. B* denotes frames for bidi- 
rectional (two-way) interframe encoding. In Fig. 2, Time 

50 js the direction in which time elapses, and in Security, a 
period during which the decoding process is halted is 
represented as a black block. Code represents the ar- 
rangement of screens in the order in which encoding is 
performed, and Display represents the arrangement of 

55 screens in the order in which of displayed. For the 
frames B, the order in which pictures are encoded differs 
from the order for the original pictures. 
[0014] Suppose that the decoding for encoded data 
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trom B A o to P 3 is halted to protect an intellectual property 
(e.g., a copyright). Since motion compensation is not 
correctly performed, as in Fig. 1 , the images in frames 
B 30 and B 31 , or B 40 and B 41 , are not correctly decoded, 
and the images in the frames are deteriorated, or the 
decoding process is hatted. 

SUMMARY OF THE INVENTION 

[0015] It is one object of the present invention to ad- 
dress the above problem, and to provide means to fa- 
cilitate an image processing apparatus and an image 
processing method for satisfactorily reproducing imag- 
es or for halting the reproduction of images to begin the 
application of the protection (e.g., the copyright) provid- 
ed for the intellectual property represented by the imag- 
es, and to provide a storage medium on which an image 
processing program is stored. 

[0016] According to one aspect of the present inven- 
tion, an image processing apparatus/method is charac- 
terized by the entering of image data, the generation of 
security data to protect the image data, the control of an 
encoding mode in accordance with the security data, 
and the encoding of the image data using the controlled 
encoding mode. 

[0017] According to another aspect of the present in- 
vention, provided is a storage medium on which an im- 
age processing program is stored, the image processing 
program comprising: 

sequence codes for an entry step for the entry of 
image data; 

sequence codes for a generation step for the gen- 
eration of security data to protect the image data; 
sequence codes for an encoding step for the control 
of an encoding mode in accordance with the secu- 
rity data; and 

sequence codes for a control step for the encoding 
of the image data using the controlled encoding 
mode. 

[0018] Other objects, features and advantages of the 
invention will become apparent during the course of the 
following detailed description, which is given while re- 
ferring to the accompanying drawings. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0019] 

Fig. 1 is a diagram for explaining the conventional 
decoding process (for data consisting of I and P 
frames) when images are protected; 
Fig. 2 is a diagram for explaining the conventional 
decoding process (for data consisting of I, P and B 
frames) when images are protected; 
Fig. 3 is a block diagram illustrating the arrange- 
ment of an image processing apparatus according 



to a first embodiment of the present invention; 
Fig. 4 is a block diagram illustrating the arrange- 
ment of a security encoding unit according to the 
first embodiment of the present invention; 
5 Fig. 5 is a diagram for explaining security encoded 
data according to the first embodiment of the 
present invention; 

Fig. 6 is a block diagram illustrating the arrange- 
ment of a video encoding unit according to the first 
10 embodiment of the present invention; 

Fig. 7 is a block diagram illustrating the arrange- 
ment of a security decoding unit according to the 
first embodiment of the present invention; 
Fig. 8 is a block diagram illustrating the arrange- 
rs ment of a video decoding unit according to the first 
embodiment of the present invention; 
Fig. 9 is a diagram for explaining the decoding proc- 
ess according to the first embodiment of the present 
invention; 

20 Fig. 10 is a diagram for explaining the security pro- 
vided for encoded data that correspond to a voice 
and a motion image according to the present inven- 
tion; 

Fig. 11 is a diagram for explaining the security that 
2B is provided for encoded data that correspond to a 
MPEG-4 multi-object according to the present in- 
vention; 

Fig. 12 is a block diagram illustrating the arrange- 
ment of a video encoding unit according to a second 

30 embodiment of the present invention; 

Figs. 1 3A, 1 3B, 1 3C, 1 3D and 1 3E are diagrams for 
explaining the change from an encoding mode for 
permitting security information to an encoding 
mode for prohibiting security information according 

35 to the second embodiment of the present invention; 
Figs. 14A, 14B, 14C, 14D and 14E are diagrams for 
explaining the change from security information for 
prohibiting an encoding mode to security informa- 
tion permitting an encoding mode according to the 

40 second embodiment of the present invention; 

Fig. 15 is a diagram for explaining the decoding 
process according to the second embodiment of the 
present invention; 

Figs. 1 6A, 1 6B, 1 6C, 1 6D and 1 6E are diagrams for 
45 explaining the change from security information 
permitting an encoding mode to security informa- 
tion prohibiting an encoding mode according to the 
second embodiment of the present invention; 
Figs. 17A, 17B, 17C : 17D, 17E and 17F are dia- 
50 grams for explaining the change from security infor- 
mation permitting an encoding mode to security in- 
formation prohibiting an encoding mode according 
to the second embodiment of the present invention; 
Figs. 18A, 18B, 18C, 18D and 18E are diagrams for 
55 explaining a change from security information pro- 
hibiting an encoding mode to security information 
permitting an encoding mode according to the sec- 
ond embodiment of the present invention; 
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Fig. 19 is a block diagram illustrating an image 
processing apparatus according to a third embodi- 
ment of the present invention; 
Fig. 20 is a block diagram illustrating a code con- 
version unit according to the third embodiment of 
the present invention; 

Fig. 21 is a block diagram illustrating a code con- 
version unit according to a fourth embodiment of the 
present invention; 

Fig. 22 is a block diagram illustrating an image 
processing apparatus according to a fifth embodi- 
ment of the present invention; 
Fig. 23 is a diagram showing the contents of a mem- 
ory according to the fifth embodiment of the present 
invention; 

Fig. 24 is a flowchart showing the image processing 
performed for the fifth embodiment of the presents 
invention; 

Fig. 25 is a flowchart showing the image processing 
performed for a sixth embodiment of the presents 
invention; and 

Fig. 26 is a flowchart showing the image processing 
performed for the sixth embodiment of the presents 
invention. 

DETAILED DESCRIPTION OF THE PREFERRED 
EMBODIMENTS 

[0020] A first embodiment of the present invention will 
now be described in detail while referring to the accom- 
panying drawings. 

[0021] Fig. 3 is a block diagram illustrating the ar- 
rangement of an image processing apparatus according 
to the first embodiment of the present invention. 
[0022] In Fig. 3, a security encoding unit 1000 en- 
codes security information used for permitting or prohib- 
iting production of all or one part of motion images. A 
video encoding unit 1001 encodes motion image data. 
[0023] In this embodiment, an explanation will be giv- 
en for the encoding of motion image data using the 
MPEG-1 encoding system. The MPEG-1 encoding sys- 
tem itself will not be explained since all the details con- 
cerning its operation are given in ISO/IEC1 1172-2. Fur- 
ther, to simplify the explanation, it is herein presumed 
that motion image data are encoded for each frame, and 
that a one-frame mode for intratrame encoding and a P- 
frame mode for employing interframe correlation are 
employed. 

[0024] A multiplexer 1002 multiplexes encoded secu- 
rity data and encoded motion image data to obtain mo- 
tion image data. And a storage device 1 003 stores the 
multiplexed motion image data. 

[0025] A communication circuit 1 005 is constituted by 
a LAN, a public line network, a radio network or broad- 
cast radio waves. Communication interfaces 1004 and 
1 006 exchange motion data with the communication cir- 
cuit 1005. 

[0026] A separator 1 007 divides the motion image da- 



ta into encoded security data and encoded motion im- 
age data. A security decoding unit 1008 decodes the 
encoded security data, and compares the decoded data 
with certification data to determine whether to instruct a 

5 halt or the resumption of the reproduction of a motion 
image. A video decoding unit 1009 decodes the encod- 
ed motion image data, and reproduces a motion image. 
A monitor 1010 displays the reproduced motion image. 
[0027] The processing for the motion image data per- 

to formed by the thus arranged apparatus will be now ex- 
plained. 

[0028] An operator (not shown) enters motion image 
data that are transmitted to the video encoding unit 
1001, and also enters, for the security encoding unit 

1$ 1000, the time to begin the application of the protection 
provided for intellectual property (a copyright in this em- 
bodiment) and the time to end the protection provided 
for the copyright, as well as security information con- 
cerning an unlocking key for unlocking the security. 

20 [0029] The security encoding unit 1000 will now be 
described in detail. 

[0030] Fig. 4 is a detailed block diagram of the security 
encoding unit 1000. 

[0031] In Fig. 4, security information is entered at an 
25 input terminal 101 . Thereafter, a security setting desig- 
nator 102 employs the security information to determine 
for which frame decoding is prohibited. A security en- 
coding unit 103 then encodes the security information 
and it is output at an output terminal 104. Based on the 
30 output of the security setting designator 102, a control 
signal generator 105 generates a control signal to con- 
trol the video encoding unit 1 001 . A timer 1 06 is used to 
synchronize the timing for the entire system, and a con- 
trol signal is output at an output terminal 107. 
35 [0032] The encoding processing performed by the 
thus arranged security encoding unit 1000 will now be 
described. 

[0033] The security setting designator 102 rearranges 
received security information following the order of the 

40 times at which the decoding prohibited commands ap- 
pear, and stores the newly arranged information. The 
security encoding unit 1 03 then encodes the security in- 
formation, and outputs the encoded data to the multi- 
plexer 1002 via the output terminal 104. 

45 [0034] The encoded security data will now be de- 
scribed in detail. 

[0035] Fig. 5 is a diagram showing the structure of the 
encoded security data. 

[0036] A CodeLength code 2001 represents the code 
50 length; an I Pcode code 2002 represents the information 
required for the certification of an intellectual property 
(a copyright in this embodiment); a SecurityStartTime 
code 2003 represents the time to begin applying the pro- 
tection provided by the copyright; and a SecurityEnd- 
55 Time code 2004 represents the time to end the applica- 
tion of the protection provided by the copyright. It should 
be noted that a plurality of code 2002 to 2004 combina- 
tions are employed if a motion image for which security 
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is desired is divided into a plurality of segments. 
[0037] In this embodiment, the encoded security data 
are multiplexed at the head of the motion image data by 
the multiplexer 1002. However, the employment of the 
multiplexing process is not thereby limited, and the en- s 
coded security data may be multiplexed with motion im- 
age data in a time sharing manner and the resultant data 
transmitted. 

[0038] Referring to Fig. 4, the security setting desig- 
nator 1 02 transmits, to the control signal generator 1 05, 10 
the time to begin to applying the protection provided by 
the copyright, and the time to end that application. The 
control signal generator 105 compares the received 
times with the time indicated by the timer 1 06, generates 
a signal for controlling the application of the protection is 
provided by the copyright, and outputs it to the video 
encoding unit 1001 via the output terminal 107. 
[0039] The video encoding unit 1001 will now be de- 
scribed in detail. 

[0040] Fig. 6 is a detailed block diagram of the video 20 
encoding unit 1001. 

[0041] In Fig. 6, image data are entered at an input 
terminal 1 21 , and a frame memory 1 22 is used to store 
an image for each frame. A subtraction unit 1 23 calcu- 
lates value differences for the pixels in each macro 25 
block. A DCT and quantization unit 124 performs a DCT 
transform and quantizes a transform coefficient. There- 
after, a texture encoding unit 125 performs entropy en- 
coding for the results obtained by the quantization. 
[0042] An inverse quantization and DCT unit 126 re- 30 
ceives a quantized transform coefficient, and performs 
an inverse quantization and an inverse DCT transform. 
An adder 1 27 adds the values for the pixels in each mac- 
ro block, and a reference frame memory 128 is used to 
store frame images to perform motion compensation. A 35 
motion compensation unit 1 29 calculates a motion vec- 
tor from data provided by frame memory 122 and the 
reference frame memory 128, and performs motion 
compensation. A motion compensation encoding unit 
1 30 generates code that is related to motion compen- 40 
sation, such as an encoding mode for a macro block or 
a motion vector, which is output by the motion compen- 
sation unit 129. 

[0043] A control signal from the control signal gener- 
ator 1 05 in Fig. 2 is input at a terminal 1 31 . A frame mode 45 
setting unit 1 32 sets a frame mode for encoding, and a 
header encoding unit 133 encodes a header include a 
frame mode. A multiplexer 1 34 multiplexes the data out- 
put by the texture encoding unit 1 25, the motion com- 
pensation encoding unit 130 and the header encoding so 
unit 1 33, and a buffer 1 35 is used for the temporary stor- 
age of the encoded data that are generated by the mul- 
tiplexer 1 34. 

[0044] A rate controller 1 36 controls the amount of 
code, and adjusts the quantization coefficient used by ss 
the DCT and quantization unit 124, and an output ter- 
minal 1 37 externally outputs the data generated by the 
multiplexer 134. 



[0045] The encoding process performed by the thus 
arranged video encoding unit 1001 will now be ex- 
plained. 

[0046] Image data for motion images are sequentially 
entered at the input terminal 121 , and are stored in the 
frame memory 122. A control signal, which for a frame 
to be encoded indicates the security information is "per- 
mission" or "prohibition", is entered at the input terminal 
131. 

[0047] First, an explanation will be given for a case 
wherein security measures are not imposed and "per- 
mission" is transmitted to the frame mode setting unit 
132. 

[0048] The frame mode setting unit 1 32 performs the 
intraf rame encoding at constant intervals, and performs 
interframe encoding for the other frames. In this embod- 
iment, every 1 5 frames an I frame is encoded in the I 
frame mode, and the other 14 frames are P frames, 
which are encoded in the P frame mode. 
[0049] The frame mode setting unit 132 transmits a 
control signal to the motion compensation unit 129, the 
frame memory 122 and the reference frame memory 
128, so that in the P frame mode, motion compensation 
is performed, or so that in the I frame mode, motion com- 
pensation is not performed. Further, the frame mode 
setting unit 132 transmits a determined frame mode to 
the header encoding unit 133, and the header encoding 
unit 1 33 generates encoded data for the header in ac- 
cordance with to the MPEG-1 specifications. 
[0050] In the I frame mode, the motion compensation 
unit 1 29 outputs a 0 for each macro block (e.g., a block 
consisting of 16 X 16 pixels). In the P frame mode, the 
motion compensation unit 129 refers to the frame mem- 
ory 122 and the reference frame memory 128 to calcu- 
late a motion vector for a macro block to be encoded, 
and employs bidirectional motion compensation to pre- 
dict a macro block that is output. 
[0051] The frame image data stored in the frame 
memory 1 22 for each macro block are transmitted to the 
subtraction unit 123. The subtraction unit 123 performs 
a subtraction procedure using the pixel values and the 
value of the macro block output by the motion compen- 
sation unit 1 29. The subtraction result of the subtraction 
unit 123 is output to the DCT and to the quantization unit 
124, which performs a DCT transform for the result. The 
DCT and the quantization unit 124 then use the quanti- 
zation coefficient to quantize the obtained value. 
[0052] The result obtained by the DCT and the quan- 
tization unit 124 is transmitted to the texture encoding 
unit 125. According to the MPEG-1 encoding specifica- 
tions, the texture encoding unit 125 encodes the encod- 
ing mode of the macro block, the quantization coefficient 
and the quantization result (entropy coding (variable 
length coding) based on Huffmann coding), and outputs 
the obtained data to the multiplexer 1 34. 
[0053] The data output by the DCT and the quantiza- 
tion unit 1 24 are transmitted to the inverse quantization 
and I DCT unit 1 26, which then performs an inverse DCT 
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transform to obtain pixel data, or error data due to mo- 
tion compensation. The adder 127 then adds the error 
data to the result obtained by the motion compensation 
unit 129. 

[0054] The pixel value obtained by the adder 127 is s 
stored in the reference frame memory 128, and the mo- 
tion vector obtained by the motion compensation unit 
1 29 is transmitted to the motion compensation encoding 
unit 130. The motion compensation encoding unit 130 
encodes the motion vector and outputs the encoded mo- io 
tion vector to the multiplexer 134. 
[0055] According to MPEG-1, the multiplexer 134 
multiplexes the outputs of the texture encoding unit 1 25, 
the motion compensation encoding unit 130 and the 
header encoding unit 1 33, and outputs the result to the is 
buffer 1 35. The encoded data stored in the buffer 1 35 
are output via the output terminal 137. In addition, the 
rate controller 136 monitors the amount of code stored 
in the buffer 135, and controls the amount of code that 
is generated by changing the quantization coefficient of 20 
the DCT and the quantization unit 1 24. 
[0056] An explanation will now be given for a case 
wherein security is imposed and the security information 
is changed from permission to prohibition. 
[0057] At this time, the frame mode setting unit 1 32 2s 
does not begin any particular operation, and normally 
sets a mode in accordance with the cycle of the I frame 
mode and P frame mode. Therefore, the other units per- 
form their operations in the same manner as they do 
when the security information denotes permission. 30 
[0058] An explanation will now be given for a case 
wherein the security is unlocked and the security infor- 
mation is changed from prohibition to permission. 
[0059] The frame mode setting unit 132 designates, 
as the I frame mode, the frame mode of the first frame 35 
that is unlocked. Therefore, the header encoding unit 
1 33 outputs encoded data indicating that the frame to 
be encoded has an I frame mode. Since the motion com- 
pensation unit 129 is not activated, motion compensa- 
tion is not performed. For the frames following the I 40 
frame, the frame mode is set in accordance with the cy- 
cle for I frames and P frames that is set in advance. 
[0060] The encoded data for a motion image output 
by the output terminal 137 are multiplexed with the en- 
coded security data by the multiplexer 1002. 45 
[0061] The multiplexed motion image data are stored 
in the storage unit 1 003, or are transmitted via the com- 
munication interface 1004 to the communication circuit 
1005. 

[0062] The processing for decoding the thus obtained so 
motion image data will now be described in detail. 
[0063] The motion image data generated above are 
received from the storage unit 1 003 or from the commu- 
nication interface 1006. The separator 1007 separates 
the input motion image data into encoded motion image ss 
data and encoded security data. The encoded motion 
image data are transmitted to the video decoding unit 
1009, and the encoded security data are transmitted to 



the security decoding unit 1004. 

[0064] The security decoding unit 1008 will now be 
described in detail. 

[0065] Fig. 7 is a detailed block diagram for the secu- 
rity decoding unit 1008. 

[0066] In Fig. 7, encoded security data is input at an 
input terminal 151 , and is decoded by a security decod- 
ing unit 152 to obtain certification information and infor- 
mation concerning the time the protection provided for 
a copyright is to be applied. A security certification unit 
153 analyzes the certification information, and deter- 
mines whether decoding should be performed or should 
be prohibited. A timer 1 54 generates a time at which to 
correctly reproduce a motion image. A control signal 
generator 155 employs the result of the determination 
made by the security certification unit 153 to generate 
a signal for controlling the video decoding unit 1009. An 
output terminal 156 is used to transmit, to the video de- 
coding unit 1005, a control signal generated by the con- 
trol signal generator 155. 

[0067] The decoding process performed by the thus 
arranged security decoding unit 1008 will now be de- 
scribed. 

[0068] The encoded security data input at the input 
terminal 151 are decoded by the security decoding unit 
152. Of the codes shown in Fig. 5, the CodeLength code 
2001 is decoded to determine the overall amount of en- 
coded data, and the IPcode code 2002, the Securit- 
yStartTime code 2003 and the Security EndTime code 
2004 are decoded to obtain infomation for each portion 
for which security is imposed. 

[0069] The information required for certification is 
transmitted to the security certification unit 153, and if 
no instruction is thereafter issued by the security certi- 
fication unit 153, the control signal generator 155 out- 
puts a control signal to the video decoding unit 1009 via 
the output terminal 156, so that the video decoding unit 
1009 will decode the encoded motion image data. 
[0070] The security certification unit 153 compares 
certification information that is entered at an operating 
console (not shown) by an operator with the certification 
information decoded by the security decoding unit 152, 
and determines whether decoding should be permitted. 
[0071] If the security certification unit 153 determines 
that decoding should be permitted, it notifies the control 
signal generator 155 of decoding is permitted. At this 
time, if the control signal generator 1 55 has already gen- 
erated a control signal to permit the video decoding unit 
1009 to decode motion image data, this control signal 
is not changed. 

[0072] If the security certification unit 1 53 determines 
that decoding should be prohibited, it outputs to the con- 
trol signal generator 155 a signal prohibiting decoding. 
The control signal generator 155 then compares the 
time output by the timer 154 with the information that is 
output by the security decoding unit 1 52 and determines 
the time at which to start to apply the protection provided 
by the pertinent copyright. When the time to apply the 



BNSDOCID: <EP 1014724A2J_> 



6 



11 



EP 1 014 724 A2 



12 



copyright protection matches the time output by the tim- 
er 154, a signal to halt the decoding of motion image 
data is output via the output terminal 156. In addition, 
the control signal generator 1 55 compares the informa- 
tion specifying the time the copyright protection is to end s 
with the time received from the timer 1 54. When the two 
times match, a signal to resume the decoding of motion 
image data is output via the output terminal 1 56. 
[0073] Thus, the video decoding unit 1009 decodes 
the encoded motion image data based on the control io 
signal received from the security decoding unit 1008. 
[0074] The video decoding unit 1009 will now be de- 
scribed in detail. 

[0075] Fig. 8 is a detailed block diagram for the video 
decoding unit 1009. is 
[0076] In Fig. 8, encoded motion image data is input 
at an image data input terminal 201 ; a control signal from 
the security decoding unit 1008 is input at a control sig- 
nal input terminal 202; and the encoded motion image 
data is stored in a buffer 203. 20 
[0077] A separator 204 reads the MPEG-1 encoded 
data from the buffer 203, and divides these data into 
header information representing the contents of code, 
motion compensation code for performing motion com- 
pensation, and data obtained by encoding a DCT trans- 25 
form coefficient. 

[0078] A header decoding unit 205, which decodes 
the header information provided by the separator 204, 
obtains information by decoding various start codes, the 
size of an image and the encoding modes of individual 30 
frames. 

[0079] A motion compensation decoding unit 206 de- 
codes the motion compensation code, the encoding 
modes and the motion vectors for individual macro 
blocks, and obtains information that is required for mo- 35 
tion compensation. A texture decoding unit 207 decodes 
codes to acquire the quantized data for the quantization 
coefficient, and the DCT transform coefficient. 
[0080] An inverse quantization and I DCT unit 208 re- 
ceives the quantized data for the quantization coefficient 40 
and the DCT transform coefficient, performs an inverse 
quantization and an inverse transform for the received 
data, and obtains pixel data, or motion compensation 
error data. 

[0081] An adder 209 adds the results obtained by the 45 
motion compensation to the error data. A reference 
frame memory 210 is used to store a reference frame 
required for motion compensation. A motion compensa- 
tion unit 211 performs motion compensation for the con- 
tents of the reference frame memory 2 1 0 in accordance so 
with the results obtained by the motion vector decoding 
unit 206. 

[0082] A display frame memory 21 2 is used to tempo- 
rarily store motion image data for the decoded frames 
before they are output. An output terminal 21 3 is used 55 
to output the motion image data to the monitor 1010. 
[0083] The decoding process performed by the thus 
arranged video decoding unit 1 009 will now be descried. 



[0084] First, when a control signal received at the con- 
trol signal input terminal 202 indicates that decoding is 
enabled, the encoded motion image data that are re- 
ceived at the image data input terminal 201 are stored 
in the buffer 203. When a control signal received at the 
control signal input terminal 202 indicates that decoding 
is prohibited, the encoded motion image data are not 
stored in the buffer 203 and are abandoned. 
[0085] The decoding process performed when the 
control signal indicates decoding is permitted will be de- 
scribed first. 

[0086] The separator 204 receives the encoded data 
from the buffer 203, and divides the encoded data into 
header information representing the contents of code, 
motion compensation code for performing motion com- 
pensation, and data obtained by encoding a DCT trans- 
form coefficient. The header decoding unit 205 decodes 
various start codes, the size of an image, and encoding 
modes for the individual frames to acquire information. 
[0087] The motion compensation decoding unit 206 
decodes the motion compensation code obtained by the 
separator 204, and the encoding modes and motion 
vectors for the individual macro blocks, and obtains in- 
formation that is required for motion compensation. For 
a P frame, the motion compensation decoding unit 206 
transmits the obtained information to the motion com- 
pensation unit 211 for the motion compensation opera- 
tion. 

[0088] The texture decoding unit 207 decodes the en- 
coded data obtained by the separator 204, and acquires 
the quantized data for the quantization coefficient and 
the DCT transform coefficient. The quantized data that 
are decoded are transmitted to the inverse quantization 
and IDCT unit 208, which then obtains pixel data, or er- 
ror data due to motion compensation. 
[0089] The adder 209 adds the error data to the data 
output by the motion compensation unit 211. If the en- 
coding mode for a macro block decoded by the motion 
compensation decoding unit 206 is intraframe coding, 
the motion compensation unit 211 outputs a value of 0 
to the adder 209. Otherwise, the motion compensation 
unit 21 1 , in accordance with the obtained motion vector, 
performs motion compensation using a reference frame 
in the reference frame memory 210, and outputs the ref- 
erence macro block data to the adder 209. 
[0090] The pixel value obtained by the adder 209 is 
stored in the reference frame memory 210 and the dis- 
play frame memory 212. And the image data stored in 
the display frame memory 213 are output via the output 
terminal 213 to the monitor 1010 to display thereon. 
[0091] An explanation will now be given for the decod- 
ing process performed when the control signal changes 
an instruction from permission to decode to decoding 
prohibited to instruct resuming of decoding. 
[0092] The entry of code in the buffer 203 is halted, 
and the operations of the header decoding unit 205, the 
motion compensation decoding unit 206, the motion 
compensation unit 211 and the texture decoding unit 
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207 are also hatted. 

[0093] Furthermore, the updating of the reference 
frame memory 210 and the display frame memory 212 
is prohibited, and as a result, the image data for the last 
frame that was decoded are output via the output termi- 
nal 212. Therefore, the last frame for which decoding 
was permitted is continuously displayed on the monitor 
1010. 

[0094] An explanation will now be given for the decod- 
ing process performed when the instruction for the con- 
trol signal is changed from decoding prohibited to de- 
coding permitted, and the resumption of decoding is in- 
structed. 

[0095] The entry of code in the buffer 203 is resumed, 
and the header decoding unit 205 is activated. Since the 
first frame is an I frame, the operation of the texture de- 
coding unit 207 is resumed, and the updating of the ref- 
erence frame memory 21 0 and the display frame mem- 
ory 212 is restarted. The operations of the individual 
units are continued to decode the I frames and the P 
frames until the control signal again indicates decoding 
is prohibited. Thus, the display on the monitor 1010 is 
updated, and the display of motion images is resumed. 
[0096] Since the decoding is resumed through the 
above described selective process sequence and the 
intraframe encoding is performed for the first frame, the 
deterioration of the frame images, or the interruption of 
the decoding process can be easily prevented. 
[0097] Suppose that, as is shown in Fig. 9, the pro- 
tection (Security) provided for a copyright is applied in 
an interval between encoded frames P 4 and P 9 , and that 
the frames can not be certified. In this case, since the 
frame P 3 located immediately before the point at which 
Security was applied is continuously displayed, and 
since the frame located immediately after the Security 
was unlocked is always an I frame, correct decoding can 
be performed beginning with the frame immediately fol- 
lowing the unlocking of the Security, and the discontinu- 
ity of images or the deterioration of images can be pre- 
vented. In addition, there is no delay (scene skipping) 
from the time the Security was unlocked until the image 
recovery was effected, and a normal image can be pro- 
vided for a user. 

[0098] In this embodiment, the image decoded imme- 
diately before decoding prohibition is imposed is output 
during the period in which the decoding is prohibited. 
However, a predetermined image may be stored in 
memory in advance, and may be output during the 
above period. 

[0099] The memory structures can be changed as 
needed in this embodiment. The encoding system for 
motion images is not limited to the MPEG-1 coding sys- 
tem; the MPEG-2 coding system can also be employed 
with a frame being used as a field, and the H.261 or the 
H.263 coding system can be employed as well. 
[0100] The format for the encoded security data, the 
certification method and the method for designating a 
portion to receive copyright protection are not limited to 



those described in the embodiment, and another format, 
another certification method and another designation 
method may be employed. For example, as is shown in 
Fig. 10 : in addition to the individual codes in Fig. 5, an 
s A/V flag 2005 may be added to signal whether a copy- 
right to be protected is for a voice or for a motion image. 
(0101] To apply this embodiment for the MPEG-4 mul- 
ti-object, an Objectcode code 2006 is added to identify 
an object, as is shown in Fig. 11, and the security is set 
10 up and encoded for each object. A plurality of video en- 
boding units are prepared, and 1-VOP is employed to 
encode the first VOP whereat the security for each ob- 
ject is unlocked, so that the security of the decoding is 
ensured and the deterioration of image quality can be 

is prevented. 

[0102] In addition, at the first portion whereat the se- 
curity is unlocked, a Group of Pictures header may be 
provided for MPEG-1 ,2, or a Group of VOPs header may 
be provided for MPEG-4. 

20 [0103] The arrangement of an image processing ap- 
paratus according to a second embodiment is the same 
as that in Fig. 1 , and only the internal arrangement and 
the processing for a video encoding unit 1001 differ. 
Therefore, only the structure and the operation of the 

25 video encoding unit 1001 will be explained for the sec- 
ond embodiment. 

[01 04] An explanation will be given for a case in which 
motbn image data is encoded by the MPEG-1 encoding 
system. Since the MPEG-1 encoding system is de- 

30 scribed in detail in ISO/IEC11172-2, no explanation for 
it will be given here. And to simplify the description, data 
are encoded for each frame, and the encoding modes 
consist of I frame modes for intraframe coding, P frame 
modes for employing interframe correlation, along a 

35 time axis, to predict a motion image from a preceding 
frame. And a B frame mode for predicting a motion im- 
age, along a time axis, from preceding and succeeding 
frames. 

[0105] The video encoding unit 1001 in the second 
40 embodiment will now be described in detail. 

[0106] Fig. 12isadetailedblockdiagramforthevideo 
encoding unit 1001 for this embodiment. 
[0107] In Fig. 12, image data are entered at an input 
terminal 301 , and frame memories 302, 303 and 304 are 
45 used to store an image for each frame. A subtraction 
unit 305 calculates a difference by subtracting the val- 
ues for pixel units for each macro block. A DCT and 
quantization unit 306 performs a DCT transform and 
quantizes a transform coefficient. A texture encoding 
50 unit 307 performs entropy encoding for the results ob- 
tained by the quantization. 

[0108] An inverse quantization and DCT unit 308 re- 
ceives a quantized transform coefficient, and performs 
an inverse quantization and an inverse DCT transform. 
55 An adder 309 adds together the values for the pixels for 
each macro block, and reference frame memories 310 
and 31 1 are used to store frame images to perform mo- 
tion compensation. A motion compensation unit 31 2 cal- 
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culates a motion vector from one of the frame memories 
302, 303 and 304 and the reference frame memory 310 
or 311, and performs motion compensation. 
[0109] A motion compensation encoding unit 313 
generates code that is related to motion compensation, s 
such as an encoding mode for a macro block or a motion 
vector, which is output by the motion compensation unit 
312. 

[0110] A control signal from the control signal gener- 
ator 105 in Fig. 4 enters an input terminal 314, while a 10 
frame mode setting unit 315 sets a frame mode for en- 
coding and a header encoding unit 31 6 encodes a head- 
er including a frame mode. A multiplexer 317 multiplex- 
es the data output by the texture encoding unit 307, the 
motion compensation encoding unit 31 3 and the header is 
encoding unit 316. A buffer 318 is used to temporarily 
store encoded data that are generated by the multiplex- 
er 317. 

[0111] A rate controller 319 controls the amount of 
code, and adjusts the quantization coefficient used by 20 
the DCT and quantization unit 306. And the data in the 
buffer 318 is output via an output terminal 320. 
[0112] The encoding process performed by the thus 
arranged video encoding unit 1001 will now be ex- 
plained. 25 
[01 1 3] Image data for motion images are sequentially 
entered at the input terminal 301 and are stored in the 
frame memory 302. A control signal, which for a frame 
to be encoded indicates whether the security informa- 
tion denotes "permission" or "prohibition", is entered at 30 
the input terminal 314. 

[0114] First, an explanation will be given for a case 
wherein the security is not applied and "permission" is 
entered in the frame mode setting unit 315. 
[01 1 5] The frame mode setting unit 31 5 performs the 35 
intraframe encoding at a constant interval, and performs 
interf rame encoding for the other frames. In this embod- 
iment, for every 1 5 frames there is an I frame that is 
encoded in the I frame mode, another 4 frames are P 
frames that are encoded in the P frame mode, and the 40 
remaining ten frames are B frames that are encoded in 
the B frame mode. One frame cycle is provided by ar- 
ranging two B frames between a P frame and an I frame 
or between two P frames. That is, the frames are ar- 
ranged as in 45 
I BBPBBPBBPBBPBBIBBP..., 
and in the arrangement, between one I frame and the 
next there are fifteen other frames. 
[01 1 6] Frame images are entered in order in the frame 
memories 302, 303 and 304, i.e., the first frame image so 
is stored in the frame memory 302, the following frame 
image is stored in the frame memory 303 and the next 
frame image is stored in the frame memory 304, follow- 
ing which the next frame image is stored in the frame 
memory 302. The remaining frame image data are then ss 
stored in order. 

[0117] The frame mode setting unit 315 determines 
the frame mode in accordance with the above frame cy- 



cle, and outputs the selected frame mode to the header 
encoding unit 316. The header encoding unit 316 gen- 
erates encoded header data in accordance with the 
MPEG-1 specifications. 

[0118] Since motion compensation need not be per- 
formed when the frame mode setting unit 315 desig- 
nates the I frame mode, a control signal is transmitted 
to the motion compensation unit 312 to instruct the out- 
put of a value of "0", without motion compensation being 
performed. Image data are read from one of the frame 
memories 302, 303 or 304 in which frames to be encod- 
ed in the I frame mode are stored. 
[0119] When the frame mode setting unit 315 desig- 
nates the P frame mode, motion compensation should 
be performed. Image data are read from one of the 
frame memories 302, 303 and 304 in which frames to 
be encoded in the P frame mode are stored, and also 
from either one of the reference frame memories 310 
and 31 1 in which the last frame image that was decoded 
is stored. These image data are transmitted to the mo- 
tion compensation unit 31 2 for the motion compensation 
operation. 

[0120] When the frame mode setting unit 315 desig- 
nates the B frame mode, motion compensation should 
be performed. Image data are read from one of the 
frame memories 302, 303 and 304 in which frames to 
be encoded in the B frame mode are stored, and also 
from both of the reference frame memories 31 0 and 31 1 . 
These image data are transmitted to the motion com- 
pensation unit 312 for the motion compensation opera- 
tion. At this time, image data encoded in the B frame 
mode are not written in the reference frame memories 
310 and 311. 

[0121] In the I frame mode, the motion compensation 
unit 31 2 outputs a value of 0 for each macro block (e.g. , 
a block consisting of 16 x 16 pixels). 
[01 22] In the P frame mode, the motion compensation 
unit 312 obtains a motion vector by referring to one of 
the frame memories 302, 303 and 304 in which a frame 
to be encoded is stored, and the reference frame mem- 
ory 310 or 311 in which the frame image is stored that 
was encoded immediately before, and outputs a macro 
block predicted by the motion compensation. 
[01 23] In the B frame mode, the motion compensation 
unit 312 obtains a motion vector by examining one of 
the frame memories 302, 303 and 304 in which a frame 
to be encoded is stored, and both of the reference frame 
memories 31 0 and 31 1 , and outputs a macro block pre- 
dicted by the motion compensation. 
[0124] For each macro block : the frame image data 
read from the frame memory are transmitted to the sub- 
traction unit 305. The subtraction unit 305 subtracts 
each pixel value from the value for the macro block out- 
put by the motion compensation unit 31 2. The result ob- 
tained by the subtraction unit 305 is then output to the 
DCT and quantization unit 306, which performs a DCT 
transform for the result. The DCT and quantization unit 
306 quantizes the obtained value by using the quanti- 
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zation coefficient. 

[01 25] The result obtained by the DCT and quantiza- 
tion unit 306 is transmitted to the texture encoding unit 
307. And according to the MPEG-1 encoding specifica- 
tions, the texture encoding unit 307 encodes the encod- 
ing mode of the macro block, the quantizatbn coefficient 
and the quantization result (entropy coding (variable 
length coding) based on Huffmann coding), and outputs 
the obtained data to the multiplexer 317. 
[0126] The data output by the DCT and quantization 
unit 306 are transmitted to the inverse quantization and 
I DCT unit 308, which then performs an inverse DCT 
transform to obtain pixel data, or error data due to mo- 
tion compensation. The error data are added to the re- 
sult obtained by the motion compensation unit 312 by 
the adder 309. 

[0127] In only the I frame mode or the P frame mode, 
the pixel value obtained by the adder 309 is stored in 
the reference frame memory 310, or in a reference 
frame memory in which the image data encoded in the 
past was stored. 

[01 28] The motion vector obtained by the motion com- 
pensation unit 312 is transmitted to the motion compen- 
sation encoding unit 313, which encodes the motion 
vector and outputs the encoded vector to the multiplexer 
317. 

[0129] According to the MPEG-1 form, the multiplexer 
317 multiplexes the values output by the texture encod- 
ing unit 307, the motion compensation encoding unit 
313, the header encoding unit 316, and outputs the re- 
sult to the buffer 318. 

[01 30] The encoded data stored in the buffer 31 8 are 
output via the output terminal 320, while the rate con- 
troller 319 monitors the amount of code stored in the 
buffer 31 and controls the amount of generated code by 
changing the quantization coefficient of the DCT and 
quantization unit 306. 

[0131] An explanation will now be given for a case 
wherein the security is applied and the information is 
changed from permission to prohibition. 
[0132] At this time, the frame mode setting unit 315 
changes a frame mode by using the frame mode proc- 
essed immediately before the indicated security infor- 
mation is changed from "permission" to "prohibition". 
This state is shown in Figs. 13Ato 13E. 
[0133] Figs. 13A to 13E are diagrams for explaining 
the changing of the encoding mode when the security 
information is changed from permission to prohibition. 
[0134] In Figs. 13A to 13E, I* represents a frame en- 
coded in the I frame mode, P* represents a frame en- 
coded in the P frame mode, and B* represents a frame 
encoded in the B frame mode. An arrow indicates a lo- 
cation whereat prohibition is designated by security in- 
formation. 

[0135] In the example in Fig. 13A, the pertinent frame 
is the head of motion image data, or the head of a Group 
of Pictures and has a closed_gop code of 1 (a flag that 
is one of the data sets included in MPEG encoded data 



and that indicates an image in a GOP can be reproduced 
separately from other GOPs), and decoding is prohibit- 
ed beginning at frame B-, , which follows frame l 0 . At this 
time, even if decoding is terminated at frame l 0 in ac- 
5 cordance with the code order, unsuccessful decoding 
does not occur, and since there is no frame for which 
deterioration of an image has occurred, the frame mode 
is not changed. 

[0136] In the example in Fig. 13B, the decoding is pro- 
10 hibited beginning at frame which follows frames l 0 
and B t . At this time, although frame P 3 is displayed after 
frame B-, in the display order, frame P 3 is located before 
frame B A in the code order, so that if frame P 3 is decod- 
ed, it may be displayed on the decoding side. Therefore, 

15 frame B-, , which was originally encoded in the B frame 
mode, is changed to encode in the P frame mode. And 
then, the order of the code following the change shown 
in Fig. 13B is obtained, and frame P 3 is not referred to. 
Even if frame P 3 is encoded in the I frame mode, the 

20 frame mode is changed in the same manner. 

[01 37] In the example in Fig. 1 3C, decoding is prohib- 
ited beginning at frame P 3 , which follows frames l 0 , B 1 
and B 2 . Since as in Fig. 1 3B frame P 3 is located before 
frame B 2 , if frame P 3 is decoded it may be displayed by 

2S the decoding side. Therefore, frames B 1 and Bg, which 
originally were in the B frame mode are changed to en- 
code in the P frame mode. Then, the order of the code 
following the change shown in Fig. 1 3C is obtained, and 
frame P 3 is not referred to. Even if frame P 3 is encoded 

30 in the I frame mode, the frame mode is changed in the 
same manner. 

[01 38] In the example in Fig. 1 3D, decoding is prohib- 
ited beginning at the frame next to frame P 3 . Since frame 
P 3 is located before frames B 1 and B^ frames B-, and 
55 B 2 may not be decoded. Therefore, frames and B^ 
which were originally in the B frame mode, are changed 
to encode in the P mode. Then, the order of the code 
following the change shown in Fig. 1 3D is obtained, and 
frames P-, and P 2 are decoded without requiring a spe- 
40 cial structure for the decoding side. Even if frame P 3 is 
encoded in the I frame mode, as is shown in Fig. 1 3E, 
the frame mode is changed in the same manner. 
[0139] An explanation will now be given for a case 
wherein the security is released and the security infor- 
ms mation is changed from prohibition to permission. 

[0140] At this time, the frame mode setting unit 315 
changes a frame mode by using the preceding and suc- 
ceeding frame modes when the indicated security infor- 
mation is changed from "permission" to "prohibition". 
50 This state is shown in Figs. 14A to 14E. 

[0141] Figs. 14A to 14E are diagrams for explaining 
the changing of the encoding mode when the indicated 
security information is changed from prohibition to per- 
mission. 

55 [0142] In Figs. 14A to 14E, an arrow indicates the lo- 
cation at which security is released and decoding is en- 
abled. 

[0143] In the example in Fig. 14 A, the pertinent frame 
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is the head of motion image data, or the head of a Group 
of Pictures having a closed_gop code of 1 , and decoding 
is permitted beginning at frame B v which follows frame 
l 0 . At this time, a problem arises in that frames B 2 
and P 3 , which follow frame l 0 , can not be correctly de- s 
coded because these frames refer to frame l 0 . There- 
fore, frame B, is changed to encode in the I frame mode. 
Then, the order of the code following the change shown 
in Fig. 14A is obtained, and frames B2 and P 3 can be 
correctly displayed without requiring a special structure io 
for the decoding side. Even if frame P 3 is encoded in the 
I frame mode, the frame mode is changed in the same 
manner. 

[01 44] In the example in Fig. 1 4B, decoding is permit- 
ted beginning at frame B 2 , which follows frames l 0 and is 
B v At this time, a problem arises in that frames B2 and 
P 3 can not be correctly decoded because these frames 
refer to frame l 0 . Therefore, frame B 2 is changed to en- 
code in the I frame mode and frame P 3 is changed to 
encode in the P frame mode. Then, the order of the code 20 
following the change shown in Fig. 1 4B is obtained, and 
frames l 2 and P 3 can be correctly displayed without re- 
quiring a special structure for the decoding side. Even 
if frame P 3 is encoded in the I frame mode, the frame 
mode is changed in the same manner. 2s 
[0145] In the example in Fig. 1 4C, decoding is permit- 
ted beginning at frame P 3 , which follows frames l 0 , B, 
and B 2 . At this time, a problem arises in that frame P 3 
can not be correctly decoded because frame P 3 refers 
to frame l 0 . In addition, frames B 1 and B 2 are located 30 
after P 3 , and another problem arises concerning the de- 
coding and displaying process to be performed for these 
frames. To obtain the correct order for the code, frames 
P 3 and Bt are changed to encode in the P frame mode, 
and frame B 2 is changed to encode in the I frame mode. 35 
Then, the order of the code following the change shown 
in Fig. 14C is obtained, and frame P 3 can be correctly 
displayed without requiring a special structure for the 
decoding side. Even if frame P 3 is encoded in the I frame 
mode, the frame mode is changed in the same manner. 40 
[0146] In the example in Fig. 14D, decoding is permit- 
ted beginning at a frame next to frame P 3 . At this time, 
since frame P 3 is located before frames B-, and B2 in the 
order of the code, frames Bj and Bg may be decoded, 
so that the encoded data B 1 and B 2 must be located 45 
before frame P 3 . Therefore, frames B 1 and B 2 are 
changed to encode in the P frame mode, and the scene 
following frame P 3 is encoded in the I frame mode. Then, 
the order of the code following the change shown in Fig. 
1 4D is obtained, and the frames following frame P 3 can so 
be correctly displayed without requiring a special struc- 
ture for the decoding side. Even if frame P 3 is encoded 
in the I frame mode, the frame mode is changed in the 
same manner. 

[0147] Or, as is shown in Fig. 14E, frame B^ may be ss 
encoded by referring to frames l 0 and P 2 , and frame B 2 
may be encoded in the P frame mode. At this time, the 
scene next to frame P 3 is encoded in the I frame mode. 



[0148] The frames are encoded in accordance with 
these encoding modes, and if there is no change in the 
encoding mode as it is related to the security, the normal 
cycle is used in encoding the frames. 
[0149] The encoded motion image data output by the 
output terminal 320 are multiplexed with the encoded 
security data by the multiplexer 1002, as in the first em- 
bodiment. 

[01 SO] Since the decoding process is the same as that 
in the first embodiment, no explanation for this process 
will be given. 

[0151] Through the above described selective proc- 
ess sequence, the frame mode is changed, due to the 
security, for a frame for which decoding is halted or re- 
sumed, or for the preceding or succeeding frame, and 
motion compensation is not performed until, in accord- 
ance with the security, a decoding prohibition or a de- 
coding permission instruction is issued. Therefore, with- 
out requiring a special structure for the decoding side, 
the deterioration of images in frames and the interrup- 
tion of the decoding operation can be easily prevented. 
[0152] Suppose that, as is shown in Fig. 15, the pro- 
tection (Security) for a copyright is applied for an interval 
between encoded frames P-, and B^, and that the 
frames can not be certified. In this case, since the frame 
P n located immediately before the Security is applied 
is continuously displayed, and since the frame located 
immediately after the Security is released is always an 
I frame, correct decoding can be performed beginning 
with the frame processed immediately after the Security 
is unlocked, and the discontinuity of images or the de- 
terioration of images can be prevented. In addition, 
there is no delay (scene skipping) between the unlock- 
ing of the Security and the image recovery, and a normal 
image can be provided for a user. 
[01 53] In this embodiment, the image decoded imme- 
diately before decoding is prohibited is output during the 
period in which decoding is prohibited. However, a pre- 
determined image may be stored in a memory in ad- 
vance, and may be output during the above period. 
[0154] The memory structures can be changed as 
needed in this embodiment. The encoding system for 
motion images is not limited to the MPEG-1 coding sys- 
tem; the MPEG-2 coding system can also be employed, 
with a frame being used as a field, and the H.261 or H. 
263 coding system can be used as well. 
[0155] In addition, even if the structure of the GOP is 
changed as in 

IBBBPBBBPBBBPBBBIBBBPBBBP... 
when decoding is halted, the frame mode is changed as 
is shown in Figs. 16A to 16E or Figs. 17A to 17E, and 
when decoding is resumed, the frame mode is changed 
for a target frame and the preceding and succeeding 
frames, as is shown in Figs. 18Ato 18E, so that motion 
compensation is not performed until, in accordance with 
the security, a decoding prohibition or a decoding per- 
mission instruction is issued. Therefore, without requir- 
ing a special structure for the decoding side, the deteri- 
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oration of images in frames and the interruption of the 
decoding operation can be easily prevented. An arbi- 
trary cycle can be employed to carry out this process. 
[01 56] Further, the B frame mode can be changed not 
only to the P frame mode, but also to the I frame mode. 
[01 57] Fig. 1 9 is a block diagram illustrating an image 
processing apparatus according to a third embodiment 
of the present invention. 

[0158] In Fig. 1 9, a storage device 400 is used to store 
encoded motion image data. In this embodiment, to sim- 
plify the explanation either the H.261 or the H.263 cod- 
ing system : which does not employ a bidirectional pre- 
dictive coding mode, is used as the system for encoding 
motion image data. This embodiment can be applied for 
an MPEG-1 or 2 coding system that does not employ a 
bidirectional predictive coding mode (a B frame mode). 
[0159] A security setting unit 401 sets security infor- 
mation for permitting or prohibiting the reproduction of 
part or all of the motion images. A code conversion unit 
402 converts encoded motion image data in accordance 
with an instruction received from the security setting unit 
401 . A security encoding unit 403 encodes security in- 
formation. A multiplexer 404 multiplexes encoded secu- 
rity data with encoded motion image data, and gener- 
ates motion image data. And a storage unit 405 is used 
for the storage of motion image data. 
[0160] The motion image data processing performed 
by the thus arranged image processing apparatus will 
now be described. 

[0161] An operator (not shown) selects encoded mo- 
tion image data stored in the storage unit 400, and in- 
puts, to the security setting unit 401 , the times to start 
and end the protection for a copyright for the encoded 
motion image data, and security information that indi- 
cates a key used to release security. 
[0162] The security setting unit 401 rearranges the re- 
ceived security information' following the order of the 
times at which the decoding prohibited commands ap- 
pear, and stores this arrangement for use when a frame 
is identified that corresponds to a motion image. The se- 
curity encoding unit 403 encodes the security informa- 
tion, and outputs the encoded security data to the mul- 
tiplexer 404. And the code conversion unit 402 converts 
code for a corresponding frame and a preceding or suc- 
ceeding frame. 

[0163] The code conversion unit 402 will now be de- 
scribed in detail. 

[0164] Fig. 20 is a detailed block diagram illustrating 
the code conversion unit 402 according to the third em- 
bodiment. 

[0165] In Fig. 20, an input terminal 451 is used to enter 
encoded motion image data, and a separator 452 sep- 
arates, into header code and encoded image data, the 
data that are encoded by the H.261 or the H.263 coding 
system. 

[0166] The header code is that which is obtained by 
encoding information representing the code character- 
istics, such as various start codes, the size of an image, 



a bit rate and a frame mode. The encoded image data 
is a code that represents a motion image, such as quan- 
tized data code, motion vector code, or the encoding 
mode for a macro block. 
5 [0167] Buffers 453 and 454 are used for the tempo- 
rary storage of codes provided by the separator 452. An 
input terminal 455 is used to enter the encoded security 
data received from the security setting unit 401 for each 
frame. A security setting instruction unit 456 determines 
io what security to set for each frame and issues instruc- 
tions to the individual units. And header decoding units 
457 and 462 decode the header code. 
[0168] Selectors 458, 459 460 and 461 select input/ 
output data in accordance with the instructions received 

is from the security setting instruction unit 456. A header 
encoding unit 463 encodes header information, and a 
decoding unit 464 decodes encoded image data to ob- 
tain a motion image. And an encoding unit 465 encodes 
motion image data to obtain encoded image data. 

20 [0169] A multiplexer 466 multiplexes generated 
codes to obtain H.261 or H.263 encoded data, and an 
output terminal 467 is used to output the encoded mo- 
tion image data obtained by the multiplexer 466. 
[0170] The encoding process performed by the thus 

25 arranged code conversion unit 402 will now be de- 
scribed. 

[0171] Motion image data are sequentially received at 
the input terminal 451, and are separated into header 
code and encoded image data by the separator 452. 

30 The header code is stored in the buffer 453, and the en- 
coded image data are stored in the buffer 454. 
[01 72] The header code stored in the buffer 453 is de- 
coded by the header decoding unit 457, and the encod- 
ing mode for a frame is obtained and output to the se- 

35 curity setting instruction unit 456. 

[0173] The security setting instruction unit 456 deter- 
mines whether security is to be applied for the individual 
frames, and analyses the frame modes of frames that, 
along a time axis, precede and follow a frame whereat 

40 the security is started and a frame whereat the security 
is ended. 

[01 74] As a result of the analysis, for a frame for which 
the frame mode must be changed, the security setting 
instruction unit 456 operates the selectors 458, 459, 460 

45 and 461 , so that the pertinent frame is passed through 
the header decoding unit 462, the header encoding unit 
463, the decoding unit 464 and the encoding unit 465. 
The security setting instruction unit 456 also transmits 
a new frame mode to the header encoding unit 463. If 

50 the frame mode does not need to be changed, input/ 
output data that pass through the above components is 
selected. 

[0175] First, an explanation will be given for a case 
wherein the security is not applied and the frame mode 
55 does not have to be changed by the security setting in- 
struction unit 456. 

[0176] The security setting instruction unit 456 issues 
instructions to the selectors 458, 459, 460 and 461 to 
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pass a frame through the header decoding unit 462 : the 
header encoding unit 463, the decoding unit 464 and 
the encoding unit 465. The encoded data stored in the 
buffers 453 and 454 are read for each frame and are 
output to the multiplexer 466. The multiplexer 466 per- s 
forms a reverse operation as the separator 452, and 
generates the same code as was entered and outputs 
it via the output terminal 467. 

[0177] An explanation will now be given for a case 
wherein the security is applied and the indicated security 10 
information is changed from permission to prohibition. 
[0178] As in the first embodiment, the frame mode is 
not changed, i.e., the same process is performed as is 
performed when the security is not applied. 
[0179] Finally, an explanation will be given for a case is 
wherein the security is released and the indicated secu- 
rity information is changed from prohibition to permis- 
sion. 

[0180] The frame mode setting unit 304 forcibly sets, 
to the I frame mode : the frame mode of the first frame 20 
after the security is released. As in the first embodiment, 
all the frames up to the next I frame are encoded in the 
P frame mode. 

[0181] That is, the security setting instruction unit 456 
issues instructions to the selectors 458, 459, 460 and 25 

461 to pass the frame through the header decoding unit 
462, the header encoding unit 463, the decoding unit 
464 and the encoding unit 465. 

[0182] Specifically, the following process is per- 
formed. 30 
[0183] Encoded data stored in the buffers 453 and 
454 are read for each frame. The header decoding unit 

462 decodes the header code, and the header encoding 
unit 463 encodes the decoded header code by replacing 

it with information concerning the frame mode. The en- 35 
coded data obtained by the header encoding unit 463 
are then output via the selector 459 to the multiplexer 
466. 

[0184] The decoding unit 464 decodes the image da- 
ta, and reproduces an image for a frame, and the en- 40 
coding unit 465 encodes the frame in the frame mode 
instructed by the security setting instruction unit 456. 
The encoded data obtained by the encoding unit 465 
are then output via the selector 461 to the multiplexer 
466. 45 
[0185] The multiplexer 466 : performing an operation 
that is the reverse of that performed by the separator 
452, multiplexes the encoded data and outputs the ob- 
tained data via the output terminal 467. The following 
process corresponds to the one performed when the in- so 
dicated security data is permission. 
[0186] Since, through the above described process 
sequence, the encoded data are converted in accord- 
ance with the state of the security, even if security data 
are added, the deterioration of images in frames on the ss 
decoding side and the interruption of the decoding op- 
eration can be easily prevented. 

[0187] The memory structures can be changed as 



needed in this embodiment. The encoding system for 
motion images is not limited to the H.261 or the H.263 
coding system, and the MPEG-1 or 2 coding systems 
can be employed. So long as the same process is per- 
formed for each object, the MPEG-4 coding system can 
also be employed. 

[0188] The arrangement of an image processing ap- 
paratus according to a fourth embodiment is the same 
as that in Fig. 1 9 : and only the internal arrangement and 
the processing performed by a code conversion unit 402 
differ. Therefore, for the fourth embodiment, only the 
structure and the operation of the code conversion unit 
402 will be explained. In this embodiment, to simplify the 
explanation, the MPEG-1 coding system, which em- 
ploys a bidirectional predictive coding mode, is used as 
the system for the encoding of motion image data. For 
this embodiment, the MPEG-2 and 4 coding systems, 
and the H.261 and H.263 coding systems can be ap- 
plied. 

[0189] Fig. 21 is a detailed block diagram illustrating 
the code conversion unit 402 according to the fourth em- 
bodiment. 

[0190] In Fig. 21, an input terminal 471 is used to re- 
ceive encoded motion image data, and a separator 472 
separates, into header code and encoded image data, 
the data that are encoded by the MPEG-1 coding sys- 
tem. 

[0191] The header code is the one obtained by en- 
coding information representing the code characteris- 
tics, such as various start codes, the size of an image, 
a bit rate, and a frame mode. The encoded image data 
is code, such as quantized data code, motion vector 
code, or the encoding mode for a macro block, that rep- 
resents a motion image. 

[0192] Buffers 473 and 474 are used for the tempo- 
rary storage of codes obtained by the separator 472. An 
input terminal 475 is used to enter for each frame the 
encoded security data received from the security setting 
unit 401. A security setting instruction unit 476 deter- 
mines what security to set for each frame and issues 
instructions to the individual units. Header decoding 
units 477 and 482 decode the header code. 
[0193] Selectors 478, 479 480 and 481 select input/ 
output data in accordance with instructions received 
from the security setting instruction unit 476. A header 
encoding unit 483 encodes header information, and a 
decoding unit 484 decodes encoded image data to ob- 
tain a motion image. And an encoding unit 485 encodes 
motion image data to obtain encoded image data. 
[0194] A multiplexer 486 multiplexes generated 
codes to obtain MPEG-1 encoded data, and an output 
terminal 487 is used to output the encoded motion im- 
age data obtained by the multiplexer 486. 
[0195] A frame mode setting unit 488 sets a frame 
mode by using the frame modes of the individual frames 
and the security information that is output by the security 
setting instruction unit 476. 

[0196] The encoding process performed by the thus 
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arranged code conversion unit 402 will now be de- 
scribed. 

[01 97] Motion image data are sequentially received at 
the input terminal 471, and are separated into header 
code and encoded image data by the separator 472. 
The header code is stored in the buffer 473, and the en- 
coded image data are stored in the buffer 474. 
[01 98] The header code stored in the buffer 473 is de- 
coded by the header decoding unit 477, and the encod- 
ing mode for a frame is obtained and output to the se- 
curity setting instruction unit 476. 
[01 99] The security setting instruction unit 476 deter- 
mines whether security is to be applied to individual 
frames, and analyses the frame modes of frames that 
precede and succeed, along a time axis, a frame 
whereat the security is started and a frame whereat the 
security is ended. 

[0200] As a result of the analysis, for a frame for which 
the frame mode must be changed, the security setting 
instruction unit 476 operates the selectors 478, 479, 480 
and 481 , so that the pertinent frame is passed through 
the header decoding unit 482, the header encoding unit 
483, the decoding unit 484 and the encoding unit 485. 
The security setting instruction unit 476 also transmits 
a new frame mode to the header encoding unit 483. If 
the frame mode does not have to be changed, the input/ 
output data that pass through the above components is 
selected. 

[0201] First, an explanation will be given for a case 
wherein the security is not applied and the frame mode 
does not have to be changed by the security setting in- 
struction unit 476. 

[0202] The security setting instruction unit 476 issues 
instructions to the selectors 478, 479, 480 and 481 to 
pass a frame through the header decoding unit 482 : the 
header encoding unit 483, the decoding unit 484 and 
the encoding unit 485. In addition, the security setting 
instruction unit 476 does not permit the frame mode set- 
ting unit 488 to be activated. 

[0203] The encoded data stored in the buffers 473 
and 474 are read for each frame and are output to the 
multiplexer 486. The multiplexer 486 performs a reverse 
operation as the separator 472, generates the same 
code as was entered, and outputs the code via the out- 
put terminal 487. 

[0204] An explanation will now be given for a case 
wherein the security is applied and the indicated security 
information is changed from permission to prohibition. 
[0205] The frame mode is changed as in the second 
embodiment. The frame mode setting unit 488 is acti- 
vated and selects one of timings shown in Figs. 1 3A to 
1 3D as the timing in accordance with which the security 
is applied, and determines the encoding modes of the 
frames preceding and succeeding the determined time. 
[0206] The security setting instruction unit 476 issues 
instructions to the selectors 478, 479, 480 and 481 to 
pass the frame through the header decoding unit 482, 
the header encoding unit 483, the decoding unit 484 and 
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the encoding unit 485. 

[0207] Specifically, the following process is per- 
formed. 

[0208] Encoded data stored in the buffers 473 and 

5 474 are read for each frame. The header decoding unit 
482 decodes the header code, and the header encoding 
unit 483 encodes the decoded header code by replacing 
the information concerning the frame mode. The encod- 
ed data obtained by the header encoding unit 483 are 

10 then output via the selector 479 to the multiplexer 486. 
[0209] The decoding unit 484 decodes image data, 
and reproduces an image for a frame, and the encoding 
unit 485 encodes the frame in the frame mode instructed 
by the security setting instruction unit 476. The encoded 

15 data obtained by the encoding unit 485 are output via 
the selector 481 to the multiplexer 486. The decoding 
and encoding are periodically performed in the desig- 
nated frame mode untill the next I frame is reached. 
[0210] The multiplexer 486 performs an operation that 

20 is the reverse of that performed by the separator 472, 
and multiplexes encoded data and outputs the obtained 
data via the output terminal 487. The following process 
corresponds to the one performed when the indicated 
security data is permission. 

25 [0211] Finally, an explanation will be given for a case 
wherein the security is released and the indicated secu- 
rity information is changed from prohibition to permis- 
sion. 

[0212] The frame mode setting unit 488 changes the 

30 frame mode in the same manner as in the second em- 
bodiment. The frame mode setting unit 488 is activated 
and selects one of timings shown in Figs. 14A to 14D 
as the timing at which the security is released, and de- 
termines the encoding mode of the frames that precede 

35 and follow the determined time. The decoding and en- 
coding are performed in the same manner as when the 
security was imposed, encoded data are multiplexed 
and the resultant data are output via the output terminal 
487. The following process corresponds to the one per- 

40 formed when the security data indicates permission. 
[0213] Since by means of the above described proc- 
ess sequence the encoded data are converted in ac- 
cordance with the state of the security, even if security 
data are added, the deterioration of images in frames 

45 on the decoding side and the interruption of the decod- 
ing operation can be easily prevented. 
[0214] The memory structures can be changed as 
needed in this embodiment. The encoding system for 
motion images is not limited to the MPEG-1 coding sys- 

50 tern, and the MPEG-2 coding system can also be em- 
ployed. So long as the same process is performed for 
each object, the MPEG-4 coding system can be em- 
ployed as well. 

[0215] Fig. 22 is a block diagram illustrating the ar- 
55 rangement of an image processing apparatus according 
to a fifth embodiment of the present invention. 
[0216] In Fig. 22, a central processing unit (CPU) 500 
provides control for the entire apparatus and performs 
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various processes, and a memory 501 provides the stor- 
age areas required by an operating system (OS) that is 
needed to control the apparatus, the software and com- 
putation processes. 

[0217] A bus 502 connects various units to exchange 5 
data and control signals; a storage unit 503 is used to 
store motion image data; and a communication circuit 
504 is constituted by a LAN, a public line network, a ra- 
dio network or a broadcasting radio network. 
[0218] A communication interface 505 receives mo- io 
tion image data from the communication circuit 504. A 
display frame memory 506 is used for the storage of 
frame data for a motion image to be displayed. And a 
monitor 507 is used to display a reproduced motion im- 
age, is 
[0219] Fig. 23 is a diagram showing the use and the 
data storage condition of the memory 501 . 
[0220] Stored in the memory 501 are: an OS, for the 
overall control of the overall apparatus and for operating 
various software programs; Systems encoder software, 20 
for encoding information for the synchronous control of 
voice and motion images; security encoder software, for 
encoding data to protect a copyright; motion image en- 
coder software, for encoding motion image data; and 
audio encoder software, for encoding sounds. 2s 
[0221] In addition, an image area for storing encoded 
image data used to refer to the data for motion compen- 
sation, and a working area for storing calculation param- 
eters are also allocated in the memory 501 . 
[0222] With the above arrangement, before the 30 
processing is begun, the Systems encoder is activated 
to monitor the synchronization of motion images with 
sounds. The motion image data that include encoded 
security data, encoded motion image data and encoded 
audio data are loaded from the storage unit 503 or the 35 
communication interface 505, and are stored in the work 
area for the memory 501 . In this explanation, the motion 
image data are encoded using the H.263 coding system; 
however, any other coding system that performs motion 
compensation can be employed. 40 
[0223] An explanation will now be given, while refer- 
ring to the flowchart in Fig. 24, for the process performed 
by the CPU 500 to encode motion image data that are 
stored in the storage unit 503. 

[0224] First, at step S01, the motion image encoder 45 
is activatedto encode the first frame in the I frame mode. 
At step S02, a check is performed to determine whether 
motion image data have been received from the storage 
unit 503. When motion image data have been received, 
program control advances to step S03. When motion im- 50 
age data have not been received, the encoding process 
is terminated. 

[0225] At step S03, the security encoder is activated 
to determine the security information for the pertinent 
motion image frame. When the indicated security infor- 5S 
mation is changed from prohibition to permission (Re- 
start), program control advances to step S04. Other- 
wise, program control goes to step SOS. 
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[0226] At step S04, the motion image encoder is ac- 
tivated to encode the pertinent motion image in the P 
frame mode in accordance with the H.263 coding sys- 
tem. Relative to the encoding for the P frame mode, the 
interf rame coding is set so that it is performed once eve- 
ry 132 frames, while macro blocks continue to remain 
at the same location. The obtained motion image data 
are multiplexed with the security information, and the re- 
sultant data are stored in the work area in the memory 
501. Program control then returns to step S02. 
[0227] At step S05, the motion image encoder is ac- 
tivated to encode the pertinent motion image in the I 
frame mode in accordance with the H.263 coding sys- 
tem. The obtained motion image data are multiplexed 
with the security information, and the resultant data are 
stored in the work area in the memory 501. Program 
control then returns to step S02. 
[0228] When the encoding process is terminated, the 
thus generated encoded data that are stored in the work 
area are read and stored in the storage unit 503, or are 
transmitted via the communication interface 305 to the 
communication circuit 504. 

[0229] As is described above, the software can be 
employed to provide one part or all of the apparatus that 
can correctly decode data without the occurrence of im- 
age discontinuity or deterioration, when the security is 
released and decoding is enabled through the above se- 
lective process sequence. 

[0230] The arrangement and operation of an image 
processing apparatus according to a sixth embodiment 
of the present invention is the same as that for the fifth 
embodiment in Fig. 22, with the exception of the process 
performed by a CPU 500 to encode motion image data 
stored in a storage unit 503. Therefore, only the encod- 
ing process will be described. 

[0231] The process performed by the CPU 500 for en- 
coding motion image data held in the storage unit 503 
will now be explained while referring to the flowcharts in 
Figs. 25 and 26. 

[0232] First, at step S1 1 , a check is performed to de- 
termine whether motion image data have been received 
from the storage unit 503. If motion image data have 
been received, program control advances to step S1 2. 
If motion image data have not been received, the en- 
coding process is terminated. 

[0233] At step SI 2, the security encoder is activated 
to determine the security information that is provided for 
the pertinent motion image frame. If the indicated secu- 
rity information is permission, program control advances 
to step S13. If the indicated security information is pro- 
hibition (Stop), program control goes to step S14. 
[0234] At step S13, the motion image encoder is ac- 
tivated to encode the first frame in the I frame mode. 
Hereinafter, the I frame mode, the P frame mode and 
the B frame mode are allocated for individual frames in 
accordance with a cycle that is determined in advance, 
and the frames are encoded. The encoded data and the 
security information are multiplexed and the resultant 
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data are stored in the work area in the memory 501 . Pro- 
gram control thereafter returns to step S11 . 
[0235] At step S 1 4, a check is performed to determine 
whether the frame mode of the last frame under the per- 
mission condition is the I frame mode. If the last frame $ 
is the I frame, program control advances to step S15. If 
the last frame is the another frame, program control 
moves to step S17. 

[0236] At step S 1 5, a check is performed to determine 
whether the frame preceding the last frame has been io 
encoded by referring to the I frame. If the I frame is not 
referred to, program control goes to step S20. If the I 
frame is referred to, program control moves to step S1 6. 
[0237] At step S16, the P frame mode is employed to 
encode frames that include those up to a P frame or an is 
I frame preceding the last frame and are located before 
the I frame preceding the last frame and are to be en- 
coded in the B frame mode. The obtained encoded data 
are multiplexed with the security information, and the re- 
sultant data are stored in the work area in the memory 20 
501 . Program control then moves to step S20. 
[0238] At step S 1 7, a check is performed to determine 
whether the frame mode of the last frame under the per- 
mission condition is the B frame mode. If the mode for 
the last frame is the B frame mode, program control ad- 25 
vances to step S18. If the mode for the last frame is an- 
other frame mode, program control goes to step S19. 
[0239] At step S1 8, the P frame mode is employed to 
encode all the frames up to the P frame or the I frame 
preceding the last frame, which include the B frame pre- 30 
ceding the last frame are to be encoded in the B frame 
mode. The encoded data are multiplexed with the secu- 
rity information, and the resultant data are stored in the 
work area in the memory 501 . Program control advanc- 
es then to step S20. 3$ 
[0240] At step S1 9, the P frame mode is employed to 
encode all the frames up to the P frame or the I frame 
before the last frame, which are located before the P 
frame preceding the last frame and are to be encoded 
in the B frame mode. Program control advances to step 40 
S20. 

[0241] At step S20, the security encoder is activated 
to determine the security information for the pertinent 
motion image frame. When the indicated security infor- 
mation is changed from prohibition to permission (Re- 45 
start), program control goes to step S22. Otherwise, pro- 
gram control goes to step S21 . 

[0242] At step S2 1 , the motion encoder software is ac- 
tivated and employs the MPEG-2 encoding system to 
encode motion image data in the individual frame so 
modes in accordance with a cycle that is determined in 
advance. The encoded data are multiplexed with the se- 
curity information, and the resultant data are stored in 
the work area in the memory 501 . 

[0243] At step S22, a check is performed to determine ss 
whether the last frame immediately before the indicated 
security information was changed from prohibition to 
permission, i.e., the last frame to which the security was 



applied, is to be encoded in the B frame mode. If the B 
frame mode is to be employed, program control goes to 
step S23. If another frame mode is employed, program 
control moves to step S24. 

[0244] At step S23, the last frame and the frames be- 
fore the last frame extending up to the P frame or the I 
frame are encoded in the P frame mode, and the encod- 
ed data are multiplexed with the security information. 
The resultant data are stored in the work area in the 
memory 501 . Program control then goes to step S26. 
[0245] At step S24, a check is performed to determine 
whether a frame immediately preceding the last frame 
to which the security was applied is to be encoded in the 
B frame mode. If the B frame mode is employed, pro- 
gram code advances to step S25. If another frame mode 
is employed, program control goes to step S26. 
[0246] At step S25, frames up to the P frame or the I 
frame before the last frame are encoded in the P frame 
mode, and the encoded frames are multiplexed with the 
security information to stored the resultant data in the 
work area of the memory 501. Then, program control 
advances to step S26. 

[0247] At step S26, the first frame immediately after 
the indicated security information was changed from 
prohibition to permission is encoded in the I frame mode. 
The encoded data are multiplexed with the security in- 
formation, and the resultant data are stored in the work 
area in the memory 501. Program control then returns 
to step S11. 

[0248] When the encoding process is terminated, the 
encoded data that have been thus generated and stored 
in the work area are read and stored in the storage unit 
503, or are transmitted via the communication interface 
305 to the communication circuit 504. 
[0249] As is described above, the frame mode is 
changed for a frame for which, due to the indicated se- 
curity, decoding is halted or resumed and for preceding 
and succeeding frames, and the motion compensation 
is not performed until, in accordance with the security, 
a decoding prohibition or permission instruction is is- 
sued. Therefore, without requiring a special structure for 
the decoding side, the deterioration of images in frames 
and the interruption of the decoding process can be eas- 
ily prevented. 

[0250] The present invention may be applied to a sys- 
tem constituted by a plurality of apparatuses (e.g., a host 
computer, an interface device, a reader, a printer, etc.) 
or to a single apparatus (e.g., a codec, a digital camera, 
etc.). 

[0251] The following system or apparatus is also in- 
cluded with in the scope of the present invention for the 
implementation of the aforementioned objectives of the 
present invention: a software program code for imple- 
menting functions of the above embodiments is loaded 
into a computer (a CPU or an MPU) in an apparatus or 
in a system that is connected to various devices to ac- 
tivate these devices, and the program code is read by 
the computer in the system or the apparatus. 
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[0252] In this case, the software program code ac- 
complishes the functions of the above described em- 
bodiment and of the program code itself, and means for 
supplying the program code to the computer, e. g. , a stor- 
age medium on which such program code is recorded, 
constitute the present invention. 

[0253] A storage medium for supplying such program 
code can be, for example, a floppy disk, a hard disk, an 
optical disk, a magneto-optical disk, a CD-ROM, a mag- 
netic tape, a nonvolatile memory card, or a ROM. 
[0254] In addition, the present invention includes not 
only a case wherein the functions in the previous em- 
bodiment can be performed when program code is read 
and executed by the computer, but also a case wherein, 
in accordance with an instruction in the program code, 
an OS (Operating System) running on the computer, or 
another application software program, the computer in- 
teracts with the program code to accomplish the func- 
tions in the above embodiments. 

[0255] Furthermore, the present invention includes a 
case wherein program code, read from a storage medi- 
um, is written in a memory that is mounted on a function 
expansion board inserted into a computer, or in a func- 
tion expansion unit connected to a computer, and in con- 
sonance with a program code instruction, a CPU mount- 
ed on the function expansion board or the function ex- 
pansion unit performs one part, or all of the actual 
processing in order to implement the functions in the 
above described embodiments. 

[0256] The format for the encoded security data, the 
certification method, and the method for designating a 
portion for the protection provided for a copyright are not 
limited to those in the above embodiments, and another 
format, another certification method and another desig- 
nation method may be employed. 
[0257] In other words, the forgoing description of em- 
bodiments has been given for illustrative purposes only, 
and is not to be construed as imposing any limitation in 
any respect. 

[0258] The scope of the invention is, therefore, to be 
determined solely by the following claims, it is not limited 
by the text of the specifications, and alterations made 
that fall within the scope of the claims reflect the true 
spirit and the intent of the invention. 



Claims 

1. An image processing apparatus comprising: 

a) input means for inputting image data; 

b) generation means for generating security da- 
ta to protect said image data; 

c) encoding means for encoding said image da- 
ta using a plurality of encoding modes; and 

d) control means for controlling said encoding 
modes of said encoding means in accordance 
with said security data. 



2. An image processing apparatus according to claim 

1 , wherein said plurality of encoding modes are at 
least an intraf rame coding mode and an interf rame 
coding mode. 

5 

3. An image processing apparatus according to claim 

2, wherein said interframe coding mode includes a 
forward predictive coding mode. 

io 4. An image processing apparatus according to claim 

3, wherein said interframe coding mode includes a 
bidirectional predictive coding mode. 

§. An image processing apparatus according to claim 
*5 - 2, wherein, when reproduction prohibition is 
changed to reproduction permission by said secu- 
rity data generation means, said control means con- 
trols said encoding modes so as to perform intra- 
frame coding for image data immediately after re- 
20 production is permitted. 

6. An image processing apparatus according to claim 

4, wherein, when reproduction permission is 
changed to reproduction prohibition by said security 

25 data generation means, said control means con- 
trols said encoding modes so as to at least perform 
forward predictive coding for image data immedi- 
ately before the reproduction was prohibited. 

30 7. An image processing apparatus according to claim 
2, wherein, when reproduction permission is 
changed to reproduction prohibition by said security 
data generation means, said control means con- 
trols said encoding modes so as to at least perform 

35 intraframe coding for image data immediately be- 
fore the reproduction was prohibited. 

8. An image processing apparatus according to claim 
1 , wherein said image data are motion image data. 

40 

9. An image processing apparatus according to claim 
8, wherein said encoding means performs an en- 
coding process that conforms to MPEG-1 . 

45 10. An image processing apparatus according to claim 
8, wherein said encoding means performs an en- 
coding process that conforms to MPEG-2. 

11. An image processing apparatus according to claim 
50 8, wherein said encoding means performs an en- 
coding process that conlorms to H.261 . 

12. An image processing apparatus according to claim 
8, wherein said encoding means performs an en- 

55 coding process that conforms to H.263. 

13. An image processing apparatus according to claim 
1 , wherein said encoding means divides one picture 
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into N objects (N is a natural number), and encodes 
each object. 

14. An image processing apparatus according to claim 
13, wherein said control means controls an encod- 
ing process for each object. 

15. An image processing apparatus according to claim 
13, wherein said security data at least include code 
for identifying said object. 

16. An image processing apparatus according to claim 
1, wherein said security data at least include code 
for certifying a copyright. 

17. An image processing apparatus according to claim 
1 , wherein image data input by said input means 
are image data encoded by using a plurality of en- 
coding modes. 

18. An image processing apparatus according to claim 

17, wherein image data input by said input means 
at least include data obtained by intraframe coding 
and data obtained by interframe coding. 

19. An image processing apparatus according to claim 

18, wherein said interframe coding is forward pre- 
dictive coding. 

20. An image processing apparatus according to claim 

19, wherein said interframe coding is bidirectional 
predictive coding. 



25. An image processing apparatus according to claim 
22, wherein, when reproduction permission is 
changed to reproduction prohibition by said security 
data generation means, said control means permits 
said encoding means to at least perform intraframe 
coding for image data immediately before the repro- 
duction was prohibited. 

26. An image processing apparatus according to claim 
17, wherein said image data input by said input 
means are data obtained by dividing one picture in- 
to N objects (N is a natural number) and by encod- 
ing each object. 



75 27. An image processing apparatus according to claim 
17, wherein said image data input by said input 
means are encoded data that conform to H.261 . 
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28. An image processing apparatus according to claim 
17, wherein said image data input by said input 
means are encoded data that conform to H.263. 

29. An image processing apparatus according to claim 
17, wherein said image data input by said input 
means are encoded data that conform to MPEG-1 . 

30. An image processing apparatus according to claim 
17, wherein said image data input by said input 
means are encoded data that conform to MPEG-2. 

31. An image processing method comprising the steps 
of: 



21. An image processing apparatus according to claim 
17, wherein said encoding means re-encodes im- 
age data input by said input means. 

22. An image processing apparatus according to claim 
17, wherein said control means employs the encod- 
ing mode of image data input by said input means 
to determine whether a re-encoding process is to 
be performed by said encoding means. 

23. An image processing apparatus according to claim 
22, wherein, when reproduction prohibition is 
changed to reproduction permission by said secu- 
rity data generation means, said control means per- 
mits said encoding means to perform intraframe 
coding for image data immediately after reproduc- 
tion is permitted. 

24. An image processing apparatus according to claim 
22, wherein, when reproduction permission is 
changed to reproduction prohibition by said security 
data generation means, said control means permits 
said encoding means to at least perform forward 
predictive coding for image data immediately before 
the reproduction was prohibited. 
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inputting image data; 

generating security data to protect said image 
data; 

controlling an encoding mode in accordance 
with said security data; and 
encoding said image data using said encoding 
mode that is controlled. 

32. A computer-readable storage medium on which an 
image processing program is stored, said image 
processing program comprising: 

a) code for an input step of inputting image da- 
ta; 

b) code for a generation step of generating se- 
curity data to protect said image data; 

c) code for an encoding step of controlling an 
encoding mode in accordance with said secu- 
rity data; and 

d) code for a control step of encoding said im- 
age data using said encoding mode that is con- 
trolled. 

33. An image processing apparatus, comprising: 
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means for receiving image data; 
means for generating security data for the re- 
ceived image data; and 

means for encoding the received image using 
at least one of a plurality of encoding modes in s 
dependence upon the generated security data! 

34. A storage device storing instructions for causing a 
programmable processing apparatus to become 
configured as an apparatus as set out in at least one 10 
of claims 1 to 30 and 33. 

35. A signal conveying instructions for causing a pro- 
grammable processing apparatus to become con- 
figured as an apparatus as set out in at least one of is 
claims 1 to 30 and 33. 
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FIG. 7 
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FIG. 19 
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